International Journal of Environment, Agriculture and Biotechnology, 5(3) 
May-Jun, 2020 / Available: https://iieab.com/ 


A descriptive analysis of the replication applied 
in aquaponic experimental studies 

Wilson A. Lennard 


University of South Australia, UniSA STEM, South Australia, 5000, Australia 
Correspondence Email address: willcnnard@gmail.com 

Abstract — A literature search was performed via SCOPUS for publications between January 2000 and 
April 2020 that contained the keywords aquaponic and hydroponic. Sixty-one articles were identified that 
stated a comparison and a form of comparative statistical analysis was performed. These articles were 
identified via the principle author, year of publication, the system type tested (coupled or decoupled 
aquaponic; irrigated nutrient solution from a separated RAS), the number of treatments tested, the number 
of replicates applied to each treatment and the region or country within which the experiment was 
performed. An experimental comparison context was assigned to each study to identify the requirement for 
replication. Sixty-one percent (61 %) of all the studies were deemed to have applied no or incorrect 
replication (no or incorrect replication: 56 % of fully recirculating system studies, 100 % of decoupled 
system studies, 86 % of irrigated RAS water studies). In terms of the comparison context, 54 % of system 
comparison studies, 100 % of solution comparison studies and 63 % of plant component associated 
comparison studies, applied no or incorrect replication. The association between study location and the 
incidence of no or incorrect replication was also determined (Europe - 71 %, USA - 80 %, South America 
- 63 %, Australia - 33 %, West Asia — 5 %, South East Asia; including China - 20 % and South Asia - 14 
%). An experimental replication decision matrix was developed to assist future aquaponic researchers in 
determining the application of correct replication and several example research articles were discussed to 
demonstrate and explain the correct and incorrect application of replication in experimental designs for 
aquaponic associated research studies. 
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I. INTRODUCTION 

Aquaponics is a technology that produces both animal 
(usually fish) and plant products and confers advantages 
such as water savings, nutrient input savings, lowered 
environmental impact and an ability for universal location 
(Lennard, 2017; Ayipio et al., 2019). A proportion of 
broader aquaculture production is done using Recirculating 
Aquaculture Systems (RAS), which are analogous to the 
fish culturing component in an aquaponic context 
(Lennard, 2017). Vegetable production is partly being done 
using hydroponic or substrate culture technologies, which 
are analogous to the plant culturing component in an 
aquaponic context (Lennard, 2017). Therefore, both fish 
and vegetables are being produced with existing water- 
culturing technologies that are analogous to aquaponics. 
This, along with the intrinsic goal of improving all 
efficiencies associated with aquaponic production 
technologies and methodologies, allows a comparative 
pathway for aquaponics in terms of fish and vegetable 


production (i.e. fish and plant growth rates may be 
compared between RAS and aquaponics and between 
hydroponics and aquaponics to establish the relative 
efficiencies of the aquaponic technique). 

Much aquaponic research is associated with establishing 
and/or improving the productive potential of the technique, 
especially via comparison to conventional hydroponic crop 
production (Ayipio et al., 2019). To measure and establish 
any perceived improvement, researchers compare the 
outputs of aquaponics (e.g. plant growth or yield) with 
either internal aquaponic variables (e.g. fish species 
cultured, fish stocking density, solution pH, 

presence/absence of additional nutrients, presence/absence 
of solution sterilisation, hydraulic parameters - flow rate or 
hydraulic retention time, planting media, etc.) or externally 
appropriate comparative technologies with known high 
productive rates for the associated organism (e.g. 

hydroponics and substrate culture - plants) (Ayipio et al., 
2019). Therefore, especially in terms of plant growth and 
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production, a proportion of aquaponic research is 
associated with experimentally comparing a proposed 
aquaponic technology variation with a known hydroponic 
culturing technique which possesses expected plant 
production outcomes that are generally considered to be 
industry best practice (Resh, 2013). 

A pivotal requirement of comparative research is the 
correct application of experimental design, which allows 
the researcher(s) to determine that the comparison they 
envisage and desire to test may be done in a context where 
the measured outcomes may be clarified and verified via 
statistical analysis (Hurlbert, 1984; Searcy-Bernal, 1994; 
Poorter & Gamier, 1996; Quinn & Keough, 2002; Ling & 
Cotter, 2003; Thorarensen et al., 2015). For comparisons 
with measured animal (fish) or plant (vegetable) productive 
outcomes, one of the most important factors that 
determines statistical rigour is the application of 
appropriate replication (Hurlbert, 1984; Searcy-Bernal, 
1994; Quinn & Keough, 2002). Replication means having 
two or more observations at a spatial or temporal scale that 
matches and repeats the application of the experimental 
treatment and are essential because biological systems are 
inherently variable (Hurlbert, 1984; Quinn & Keough, 
2002). Also, and importantly, replication assists to avoid 
confounding treatment differences with other systematic 
differences between experimental units (Hurlbert, 1984; 
Quinn & Keough, 2002). 

An important factor in ensuring that replication is adequate 
and correct in aquaculture or hydroponic studies is 
determining what constitutes an independent experimental 
unit (Tlusty, 2010; Thorarensen et al., 2015)? In a RAS 
fish culture context, an understanding of, and respect for, 
what is being tested is very important because it dictates 
the experimental design. For example, the experimental 
design to test a particular overall RAS technical design 
(Design A) against another overall RAS technical design 
(Design B), when fish growth is the measured comparative 
parameter, is different from what is required when fish 
feed variations (e.g. Feed A vs Feed B) are being compared 
within a RAS context and again, fish growth is the 
measured comparative parameter. In both instances, fish 
growth is the measured outcome used for comparison. 
However, in the first case (RAS technical design), the test 
variable is the RAS itself; in the second case (the feed 
being fed to the fish within the RAS) the test variable is the 
fish feed. 

Replication is adequate when the variable is repeated in an 
environment where other variables are completely negated 
or minimised (Hurlbert, 1984). Replication is inadequate 
(or “pseudo”) if samples come from a single experimental 
unit (Hurlbert, 1984). In the first example case, both entire 


RAS’s (A and B) must be repeated (or replicated). This is 
because the test is to determine if one RAS performs better 
than the other and what is therefore, being compared or 
tested, are the RAS designs themselves. This means a 
minimum of three RAS’s of Design A and three RAS’s of 
Design B are required. This replication of the entire RAS 
design allows the determination of acceptable absence of 
variability within that RAS design (e.g. if the fish growth 
results from all three RAS A designs are statistically 
similar, the within RAS A inherent variability is low and 
therefore, it is acceptable that any of the three replicates of 
RAS A represents the RAS A design truly, and that the 
results from all three replicates of RAS A may be pooled to 
make a common data set). In the second case, because the 
effect of the fish feed on fish growth is being tested, it is 
adequate to apply individual fish tanks as the repeatable 
experimental unit and therefore, one RAS is suitable (as 
long as it contains three or more individual fish tanks for 
each treatment - fish feed, being tested). In fact, a 
centralised filtration system that treats all the water from 
all experimental units (fish tanks) is a better design, 
because this removes the filtration unit as a potential 
variable in the comparison (i.e. if each fish tank has its own 
filter, or if each treatment of fish feed variation, of three 
fish tanks, have their own filter, how can it be determined 
if it is the fish feed or the filter that is causing any fish 
growth differences detected?) (Tlusty, 2010). 

Determining the replication requirement in a hydroponic 
context is similar and replication is inadequate, again, if all 
samples arise from one experimental unit (Hurlbert, 1984). 
If one wishes to compare hydroponic system Design A to 
hydroponic system Design B, in terms of plant growth 
outcomes, three replications of each system is required 
(three hydroponic systems of Design A and three 
hydroponic systems of Design B). If one wishes to 
compare the effect of different planting media (e.g. coir, 
perlite, vermiculite) on a plant growth outcome, a single 
hydroponic system is adequate, with replicated planting 
units (e.g. at least three NFT channels or three DWC beds, 
etc.) (Gomez & Gomez, 1984; Poorter et al., 2012; Gupta 
et al., 2015). 

Many contexts exist within RAS, hydroponics and 
aquaponics that may be compared, and it is apparent that 
because aquaponics includes two technologies (RAS and 
hydroponics) and two productive organisms (fish and 
plants), the contexts become more complex. For example, 
one may determine to compare within a system context 
(e.g. aquaponic system vs hydroponic system), a delivered 
nutrient solution context (e.g. aquaponic nutrient solution 
vs hydroponic nutrient solution), an aquaponic system 
plant component context (e.g. deep water culture vs NFT 
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vs media culture), an aquaponic system plant growing 
media context (e.g. no media, coco fibre, gravel, sand, 
perlite, vermiculite, mixtures of all of these media, etc.), an 
aquaponic system fish species context (e.g. Tilapia vs 
Trout vs Catfish, etc), mechanical or hydraulic contexts 
(e.g. applied filtration, hydraulic retention times, flow 
rates), chemical contexts (nutrient solution strength, 
nutrient solution mixture, presence/absence of plant growth 
promoters, etc.) and on it goes! Similarly, different 
parameters may be used as the measure of performance or 
comparison (e.g. fish growth, SGR, feed conversion ratio, 
etc., plant growth, solution nutrient removal or 
accumulation, etc.). These myriad contexts have different 
and unique requirements in terms of the replication applied 
within the overall experimental design. Therefore, a 
complex matrix exists for the researcher to navigate to 
determine what replication is appropriate to confer 
statistical confidence in the results produced and to ensure 
that the inferences made from those results are valid and 
ultimately, applicable to broader situations (e.g. 
commercial applications of the technology). 

RAS fish and hydroponic plant production are existing 
technologies with decades of applied research (Timmons et 
al., 2002; Resh, 2013). Therefore, techniques in 
experimental design, replication and statistical analysis 
have been well-developed and proven within these fields 
(Tlusty, 2010; Poorter et al., 2012; Gupta et al., 2015). 
Both RAS and hydroponics research demonstrate that 
replication determination is highly dependent upon the 
variable being compared (Tlusty, 2010; Poorter et al., 
2012; Gupta et al., 2015). Determination of the context of 
the comparison within both RAS and hydroponics is well 
established and should be used as a key driver for similar 
determinations applied to aquaponic research. In this way, 
within aquaponics experimentation, context determination 
enables the identification of which factor requires 
replication within the experimental design and is pivotal to 
a statistically valid outcome, whilst enabling the 
conservation of valuable resources (funds, space, 
apparatus, consumables, etc.) within the often constrained, 
experimental environment (Thorarensen et al., 2015). 

The aim of this study was to analyse the context of 
comparison and the associated replication applied to that 
context, in a randomly selected cohort of peer reviewed, 
published, scientific articles about aquaponic technology 
which applied scientific comparisons. The studies were 
examined, and the context of comparison determined. The 
replication applied was then examined to determine if it 
was appropriate for the context of the comparison. The 
outcomes of the analysis were then used to determine the 
frequency of inappropriate replication applications within 


several sub-sets of aquaponic research. The analytical 
outcomes were then used to develop a decision matrix for 
researchers in an attempt to improve the application of 
appropriate replication in experimental designs for 
aquaponics experimental research that could produce valid, 
comparative statistical outcomes. 

II. MATERIALS AND METHODS 

A literature search was performed between the dates 
12 th May 2020 and 19 th May 2020, via SCOPUS. Sixty-one 
articles (excluding reviews and editorials) published in 
peer reviewed journals were identified between January 
2000 and April 2020 that contained the keywords 
aquaponic and hydroponic, stated a comparison for some 
form of plant growth or production parameter, stated the 
inclusion of replication and stated that a form of 
comparative statistical analysis was performed. 

The identified studies were tabulated to include the name 
of the first author, the year of publication, the system 
type(s) applied (see below), the number of treatments 
applied, the number of replications applied per treatment, 
the region or country within which the experiments were 
performed and the applied context of comparative analysis. 

In terms of the system type, a recirculating aquaponic 
system was identified as a system that contained fish (in a 
fish tank), a form of filtration (mechanical, biological or 
both) and a hydroponic component where the system water 
recirculated between the fish and plant components 
perpetually. A hydroponics system irrigated with RAS 
water was identified as a system not containing fish, 
whereby a hydroponic component was irrigated with water 
that originated from a separate RAS that was not part of the 
experimental design (i.e. the RAS was not connected to the 
hydroponic component of the experimental design in any 
hydraulic form or sense, but RAS water was used to fill the 
sump or nutrient tank of the experimental hydroponic 
component). A decoupled system was identified as a 
system that contained fish (in a fish tank or fish component 
with associated filtration) and a hydroponic component 
whereby the water flowed from the fish component to the 
hydroponic component but did not return to the fish 
component from the hydroponic component. 

The number of treatments applied was identified from the 
article text with a primary relation to a plant production 
outcome via an associated parameter of identified testable 
difference. For example, hydroponic vs aquaponic, a plant 
nutrient solution difference (pH, nutrient concentration, 
nutrient mixture, hydraulic retention time), type of plant 
growing component, type of plant substrate applied, etc. 
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The number of replications applied per treatment was 
identified from the article text, the written description of 
the applied experimental set-up and design and any 
associated diagrammatic or pictorial description of the 
experimental set-up and design. The comparison parameter 
applied was identified and assigned to one of three 
categories; a system comparison (i.e. where different 
system types are compared to each other - e.g. aquaponic v 
hydroponic, variations in fish component configuration, 
variations in fish species cultured, variations in flow rates 
applied, differences in fish management, etc.), a nutrient 
solution comparison (i.e. where different nutrient solutions 
are compared, variations in solution chemical parameters, 
variations in nutrient solution concentrations, variations in 
nutrient solutions mixture, etc.), or a plant component 
comparison (i.e. where different plant component 
configurations, designs, substrates, etc. were applied). 

A series of descriptive statistics were identified and 
tabulated to include the total number of studies identified, 
the number of fully recirculating system designs, the 
number of decoupled system designs and the number of 
irrigated RAS water designs, the number of system 
comparisons, the number of solution comparisons and the 
number of plant component comparisons, the number and 
percentage of correctly replicated studies, the number and 
percentage of incorrectly replicated studies and a break¬ 
down of the same descriptive statistics based on the region 
or country of location of the research. 

The above outlined information was then used to develop 
an experimental design decision matrix so that future 
researchers have an available primer for correct 
experimental design in an aquaponic research context. In 
addition, several studies were isolated, examined and 
discussed in terms of the applied experimental design and 
replication to provide examples of valid and invalid 
application of replication, to support and illustrate the 
developed decision matrix. 

III. RESULTS 

Study Analysis and Descriptive Statistics: 

Table 1 outlines all sixty-one identified studies included in 
the analysis (by primary author), and includes the year of 
publication, the system type tested (Recirculating or 
Coupled, Hydroponics Irrigated with RAS Water and 
Decoupled), the number of treatments tested, the number 
of replications applied to each treatment, the region or 
country where the experiments were conducted and the 
comparison context. 


A series of descriptive statistics for the identified studies 
are presented in Table 2. Of the sixty-one identified 
studies, fifty-two were identified as testing recirculating 
(coupled) aquaponic systems, two were identified as 
testing decoupled aquaponic systems(where fish were 
present in a connected, hydraulic context) and seven were 
identified as testing a variation of irrigating water to a 
hydroponic component that was sourced from a separate, 
operating RAS. 

Forty-six studies were identified as comparing the whole or 
complete aquaponic systems (i.e. system comparisons), 
seven studies were identified as comparing nutrient 
solution(s) (i.e. solution comparisons) and eight were 
identified as comparing a variation associated with a plant 
component (i.e. plant component comparisons). Of the 
forty-six system comparisons, twenty-one (46 %) were 
identified to have applied correct replication of the 
treatments tested and twenty-five (54 %) were identified to 
have applied incorrect replication of the treatments tested. 
Of the seven solution comparisons, zero (0 %) were 
identified to have applied correct replication of the 
treatments tested and seven (100 %) were identified to 
have applied incorrect replication of the treatments tested. 
Of the eight plant component comparisons, three (37 %) 
were identified to have applied correct replication of the 
treatments tested and five (63 %) were identified to have 
applied incorrect replication of the treatments tested. 

Of the fifty-two recirculating (coupled) aquaponic systems 
studied, twenty-three (44 %) were identified to have 
applied correct replication of the treatments tested and 
twenty-nine (56 %) were identified to have applied 
incorrect replication of the treatments tested. Of the seven 
hydroponic components irrigated with RAS water and/or 
hydroponic nutrient solution controls, one (14 %) was 
identified to have applied correct replication of the 
treatments tested and six (86 %) were identified to have 
applied incorrect replication of the treatments tested. Of the 
two decoupled aquaponic systems studied, zero (0 %) were 
identified to have applied correct replication of the 
treatments tested and two (100 %) were identified to have 
applied incorrect replication of the treatments tested. 

Overall, of the sixty-one identified studies, twenty-four (39 
%) were identified to have applied correct replication of 
the treatments tested and thirty-seven (61 %) were 
identified to have applied incorrect, or no, replication of 
the treatments tested. 

Table 2 also contains a regional breakdown of descriptive 
statistics which includes all of those outlined above, within 
a regional context. Importantly, this outlines the percentage 
of studies performed per region that applied incorrect or no 
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Table 1: Author, year of publication, type of system comparison performed, treatment number, replicate number, region and 
comparison parameter for the identified aquaponic studies, 2000 - 2020. 


Author 

Year 

System Type Treatments 

Replicates 

Region 

Comparison Context 

Alcatraz et al. 

2018 

Recirculating 

2 

3 

Chile 

System (AP v HP) 

Alvarez-Garcia, et al. 

2019 

RAS irrigated 

5 

1 

Europe 

Solution - nutrients 

Anderson, et al. 

2017 

Recirculating 

3 

1 AP; 2 HP 

USA 

System (pH) 

Blanchard, et al. 

2020 

RAS irrigated 

4 

1 

USA 

Solution (pH) 

Blidariu, et al. 

2013 

Recirculating 

2 

1 

Europe 

System (AP v Soil) 

Calone, et al. 

2019 

RAS irrigated 

4 

1 AP: 3 HP 

Europe 

Solution (Nutrients) 

Castillo-Castellanos, et al. 2016 

Recirculating 

4 

3 

Mexico 

System (AP v HP) 

Dediu et al 

2012 

Recirculating 

2 

1 

Europe 

System (HRT) 

Delaide, et al. 

2017 

Recirculating 

2 

1 

Europe 

Plant Comp (Ebb & flow v DWC) 

Delaide, et al. 

2016 

RAS irrigated 

3 

1 

Europe 

Solution (HP v AP) 

Delaide, et al. 

2019 

RAS irrigated 

2 

1 

Europe 

Solution (HP v AP) 

Effendi etal 

2017 

Recirculating 

3 

3 

Indonesia 

System (suppl bact) 

Estrada-Perez, et al. 

2018 

Recirculating 

3 

3 

Mexico 

System (Fish density) 

Goddek & Vermeulen 

2018 

RAS irrigated 

2 

1 

Europe 

Solution (Nutrients) 

Graber &Junge 

2009 

Recirculating 

3 

1 

Europe 

System - (AP v HP v Soil) 

Hundley et al 

2018 

Recirculating 

4 

1 

Brazil 

System (Fish density) 

Husain, et al. 

2014 

Recirculating 

5 

3 

India 

System (Fish density) 

Jordan, etal. 

2018 

Recirculating 

4 

1 

Brazil 

System (AP v HP) 

Knaus & Palm (a) 

2017 

Recirculating 

2 

1 

Europe 

System (Carp v Tilapia) 

Knaus & Palm (b) 

2017 

Recirculating 

2 

1 

Europe 

System (Catfish v Tilapia) 

Lennard & Ward 

2019 

Recirculating 

2 

1 

Australia 

System (AP v HP) 

Lennard & Leonard 

2006 

Recirculating 

4 

3 

Australia 

System (Plant component) 

Lennard & Leonard 

2004 

Recirculating 

2 

3 

Australia 

System (Recip v constant) 

Maucieri, etal. 

2019 

Recirculating 

3 

3 

Europe 

System (Fish density) 

Medina, etal. 

2016 

Recirculating 

2 

3 

USA 

System (Feed: fishmeal v veg) 

Monsees, et al. 

2017 

Reci vs Decoup 

3 

1 

Europe 

System (Coupled v decoupled) 

Monsees, et al. 

2019 

RAS irrigated 

3 

3 

Europe 

System (Steril v unsteril) 

Moya et al 

2016 

Recirculating 

3 

1 

Mexico 

System (3 diff plants) 

Ngo, et al. 

2017 

Recirculating 

3 

1 

Vietnam 

System (1 - flow; 2 - fish dens) 

Nicoletto, et al. 

2018 

Recirculating 

3 

3 

Europe 

Solution (APvHP) 

Nozzi, et al. 

2018 

Recirculating 

4 

1 

Europe 

System (AP v HP) 

Nuwansi, etal. 

2017 

Recirculating 

4 

3 

India 

System (Fish ratios) 

Palm, et al. 

2014 

Recirculating 

2 

1 

Europe 

System (Catfish v Tilapia) 

Perez-Urrestarazu, etal. 2019 

Recirculating 

4 

1 

Europe 

System (raft v DWC media) 

Pinho, et al. 

2018 

Recirculating 

2 

1 

Brazil 

System (Fish: Pacu v Tilapia) 

Rafiee, et al. 

2018 

Recirculating 

2 

3 

Malaysia 

System (AP v AP complim) 

Rayhan, et al. 

2018 

Recirculating 

4 

3 

Bangladesh 

System (Fish density) 

Roosta 

2014 

Recirculating 

2 

3 

Iran 

Plant (K v no-K) 

Roosta & Afsharipoor 

2012 

RAS irrigated 

2 

1 

Iran 

System (HP v AP) 

Roosta & Hamidpour 

2011 

Recirculating 

8 

3 

Iran 

System (HP v AP) 

Roosta & Mohsenian 

2012 

Recirculating 

4 

3 

Iran 

System (HP v AP) 

Saha, et al. 

2016 

Recirculating 

2 

4 

USA 

System (HP v AP) 

Sace, & Fitzsimmons 

2013 

Recirculating 

2 

1 

USA 

System (Prawn: present v absent) 

Salam, et al. 

2014 

Recirculating 

3 

1 

Bangladesh 

Plant component (Substrates) 

Saufie, et al. 

2015 

Recirculating 

2 

3 

Malaysia 

System (AP v HP) 

Schmautz, et al. 

2016 

Recirculating 

3 

3 

Europe 

Plant Compon. (NFT v DWC v Drip) 

Shete, et al. 

2017 

Recirculating 

3 

3 

India 

Plant Component (Rock v Raft) 

Shete, et al. 

2016 

Recirculating 

3 

3 

India 

System (HRT) 

Shete, et al. 

2013 

Recirculating 

4 

3 

India 

System (Fish density) 

Silva, et al. 

2018 

Recirculating 

2 

1 

Mexico 

Plant Compon. (Raft v Dyn. RAFT) 

Simeonidou, et al. 

2012 

Recirculating 

3 

1 

Europe 

System (Fish densities) 

Sirakov 

2020 

Recirculating 

2 

1 

Europe 

Plant component (RAFT v Media) 

Suhl, etal. 

2018 

Decoupled 

2 

1 

Europe 

System (AP v HP) 

Suhl, etal. 

2016 

Decoupled 

2 

1 

Europe 

System (AP v HP) 

Vandam, et al 

2017 

Recirculating 

3 

1 

USA 

System (Hydro - 2 v AP) 

Velichkova, et al. 

2019 

Recirculating 

2 

1 

Europe 

Plant component (Raft v Media) 

Weilgosz, et al. 

2017 

Recirculating 

4 

1 

USA 

System (HP v AP; Ster v Non-steril) 

Wilson, et al 

2017 

Recirculating 

2 

1 

USA 

System (HP v AP) 

Yang & Kim 

2019 

Recirculating 

3 

2 

USA 

System (HP v AP) 

Yang, et al. 

2020 

Recirculating 

3 

2 

USA 

System (Flow rates) 

Zou, et al. 

2016 

Recirculating 

3 

3 

China 

System/Solution (pH) 
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replication to their experimental designs; Europe - 71 %, West Asia - 25 %, South East Asia (including China) - 20 

USA - 80 %, South America, 63 %, Australia - 33 %, % and South Asia - 14 %. 

Table 2 (A&B): (A) Descriptive statistics (Total studies, total studies with Correct Replication, % Correct Replication, total 
studies with Incorrect Replication, % Incorrect replication) and (B) breakdown per region (Europe, USA, South America - 
includes Central America, Australia, West Asia, South East Asia - includes China, South Asia) for the identified aquaponic 
studies 2000 - 2020. 


Parameter 

Total 

Correct 

Replication. 

% Correct 

Incorrect 

Replication. 

% Incorrect 

Number of studies identified 

61 

24 

39 

37 

61 

Fully Recirculating 

52 

23 

44 

29 

56 

Proper decoupled (fish present) 

2 

0 

0 

2 

100 

Irrigated RAS water (no fish) 

7 

1 

14 

6 

86 

System Comparisons 

46 

21 

46 

25 

54 

Solution Comparisons 

7 

0 

0 

7 

100 

Plant Component Comparisons 

8 

3 

37 

5 

63 

No. Incorrect Replications 




37 


% Incorrect Replications 




61 



Parameter 

Europe 

USA 

Sth America 

Australia 

West Asia 

SE 

Asia/China 

South 

Asia 

Number of studies identified 

24 

10 

8 

3 

4 

5 

7 

Fully Recirculating 

17 

9 

8 

3 

3 

5 

7 

Proper decoupled (fish present) 

2 

0 

0 

0 

0 

0 

0 

Irrigated RAS water (no fish) 

5 

1 

0 

0 

1 

0 

0 

System Comparisons 

13 

9 

7 

3 

3 

5 

5 

Solution Comparisons 

7 

1 

0 

0 

0 

0 

0 

Plant Component Comparisons 

4 

0 

1 

0 

1 

0 

2 

No. Incorrect Replications 

17 

8 

5 

1 

1 

1 

1 

% Incorrect Replications 

71 

80 

63 

33 

25 

20 

14 


Experimental Replication Decision Matrix: 

Figure 1 outlines a Decision Matrix to enable the 
determination of valid replication for the devised 
experimental comparisons in an aquaponic context 
associated with a plant growth or production measured 
outcome being statistically compared. A difference in a 
plant production or growth parameter is the most often 
applied comparison to determine an assumed difference 
within an aquaponic system context (e.g. comparison of 
one aquaponic system to another - coupled v decoupled 
aquaponics, etc.) or via a comparison to an external, 
differentiated context (e.g. comparison of aquaponics to 
hydroponics). 

The outlined Decision Matrix (Figure 1) will not be 
applicable to each and every experiment performed using 
aquaponic systems. However, in the context of using a 
plant measure outcome as the determinant of establishing a 


difference in treatments where aquaponic systems are 
being tested, it is expected it should account for most 
situations. 

IY. DISCUSSION 

Study Analysis and Descriptive Statistics 

It must be noted that a specific search technique was 
applied to identify studies suitable for this analysis (see 
Materials and Methods section). While the author believes 
a relatively high proportion of aquaponic studies using a 
plant growth measure as the comparative determinant (or 
one of the comparative determinants) were identified and 
included, it is acknowledged there will be studies that were 
not identified or included here. Ayipio et al. (2019) 
performed a recent meta-analysis of crop 
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Determine the Comparison 

(what is being compared?] 


NZ 

Identify the Comparison Context 

(most will be a System, Solution or Plant Component context] 


Entire System Context 

Aquaponic v Hydroponic 

Coupled v Decoupled AP 
Fish sp. comparisons 
Fish density comparisons 
Fish feeding comparisons 

Fish: Plant comparisons 
pH comparisons 
Flow rate comparisons 
Hydraulic retention times 
Sterilisation/no Sterilisation 
Filtration comparisons 


V 7 

Nutrient Solution Context 
(no fish in system) 

Aquaponic v Hydroponic 

Nutrient soln. comparisons 
pH comparisons 
Flow rate comparisons 
Supplementation 
comparisons 


Plant Component Context 

Hydroponic type 
comparisons 
Substrate comparisons 
Media comparisons 
Foliar spray comparisons 
Pruning comparisons 


Fig.l: Decision matrix for determining required replication in an aquaponic experimental context. 


yields in studies comparing conventional hydroponics to 
aquaponics and identified twenty-seven studies within a 
year range of 2009 to 2018, even though no publication 
date limitation was placed on the search range. However, 
the current study identified two studies that compared 
aquaponics to hydroponics (Lennard & Leonard, 2004; 
Lennard & Leonard, 2006) that were not identified and 
included by Ayipio et al. (2019); demonstrating that 
searches, no matter how directed the applied criteria, can 
miss relevant studies. Therefore, the sixty-one studies 
identified were considered representative of the studies 
using the stated search criteria for the time range (2000 to 
early 2019). 

There have been a number of authors who have conducted 
comparative aquaponic experiments or trials in a number 
of countries across most continents (Table 1). While some 
authors claim that decoupled aquaponic system designs are 
important and the most appropriate to future commercial 
adoption of aquaponics (Delaide et al., 2016; Goddek et al., 
2016; Monsees et al., 2017; Goddek & Vermeulen, 2018; 


Delaide et al., 2019; Goddek et al., 2019; Blanchard et al., 
2020), only two out of the sixty-one (3 %) studies 
identified, applied true decoupled designs (i.e. a system 
with a direct hydraulic linkage from a fish component 
containing fish to the plant component), and only one study 
(2 %) directly compared a true decoupled aquaponic 
system with a coupled (or fully recirculating) aquaponic 
system (Monsees et al., 2017). Seven studies compared a 
plant growth measure of some form using nutrient 
solutions only (i.e. there was no direct hydraulic linkage 
from a fish component containing fish to the plant 
component), with several arguing this approach was a valid 
analogue for true aquaponics (Delaide et al., 2016; Goddek 
& Vermeulen, 2018; Nicoletto, et al., 2018). This 
demonstrates the decoupled approach only represented 15 
% of the total identified studies. In addition, seven of the 
nine (78 %) decoupled or nutrient solution studies were 
located within Europe; suggesting that Europe is the 
epicentre of decoupled aquaponic design research. 
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The descriptive statistics (Table 2) illustrate several 
interesting points associated with aquaponic system 
research and experimentation. The vast majority of studies 
(fifty-two of sixty-one studies - 85 %) used a form of fully 
recirculating (coupled) aquaponic system and these studies 
applied replication correctly to the highest proportion (44 
%). Only two studies used true decoupled aquaponic 
systems and none (0 %) of these studies applied correct 
replication. Of the seven studies that irrigated collected 
RAS water as a nutrient solution (i.e. no fish present in a 
direct hydraulic link to the plant component), only one (14 
%) applied correct replication. 

Overall, 61 % of all studies applied replication incorrectly 
or not at all. Hulbert (1984) points out that 48% of the 
ecological field studies he examined which applied 
inferential statistics (e.g. identification of significant 
differences) contained pseudo replication. Thorarensen et 
al. (2015) points out how little attention is given to 
experimental design in aquaculture fish growth 
experiments, especially in terms of the number of required 
rearing units for adequate treatment replication and Araujo 
(2008) argued there is a lack of appreciation of basic 
statistics in aquaculture experiments. Ruohonen (1998) 
also points out that a lack of tank replication in fish 
aquaculture experiments is common, leading to a lack of 
independent statistical outcomes or pseudoreplication. 
Raudonius (2017) found that almost 55% of the articles 
associated with crop research he examined applied 
experimental design and statistical analysis incorrectly and 
Kramer et al. (2016) found a similar result of almost 50% 
for crop studies. Therefore, the outcome of the current 
study is not unexpected, and while researchers of 
aquaponics apply statistical analysis to their work, it 
appears it is more likely than not that it is applied 
incorrectly. 

The result of the current study for incorrect replication 
application (61 %) in aquaponic studies should be a major 
concern to the aquaponic research community as inferences 
are being made based on invalid experimental designs and 
statistical outcomes that do not support those inferences. 
To this end, determinant arguments are then being made 
based on these incorrect designs and analyses. Finally, 
these interpretations are being used to inform commercial 
applications of aquaponic technology. 

Table 2 also outlines the relative frequency of the 
descriptive statistics calculated for aquaponics research by 
region. Europe performed the majority of research into 
comparative aquaponics with plant measure outcomes, 
accounting for twenty-four (39 %) of all the identified 
studies (Table 1). The USA (ten - 16 %) and South 
America (eight - 13 %) were next, with South Asia (Indian 
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sub-continent). South East Asia (including China), West 
Asia (Iran, Iraq, Middle East) and Australia following. The 
most frequent experimenters (Europe, USA & South 
America) also demonstrated the highest incidence of 
incorrect or no application of replication in their studies 
(71, 80 & 63 % respectively). This demonstrates that the 
regions performing the most aquaponic-associated, 
comparative research, show the lowest application rate of 
correct replication within their experimental designs. 

This trend of incorrect application of replication within 
aquaponic research is similar to what is seen within 
aquaculture and crop science fields, which also 
demonstrate trends towards pseudo replication, no 
replication and inappropriate statistical analysis (Hulbert, 
1984; Ruohonen, 1998; Araujo, 2008; Thorarensen et al., 
2015; Kramer et al., 2016; Raudonius, 2017). Precedent 
does exist within aquaponic research for early (Lennard & 
Leonard, 2004; Lennard & Leonard, 2006) and mid- 
twenty-first century (Roosta & Hamidpour, 2011; Roosta 
& Mohsenian, 2012; Shete, et al., 2013) laboratory, 
comparative aquaponic trials and experiments that applied 
correct experimental replication and for later work that 
referenced early articles for correct experimental design 
direction (Medina, et al., 2016). Therefore, it is difficult to 
understand why such a high rate of incorrect replication is 
still being applied in the field of aquaponics? 

An example of industry informing inferences is 
demonstrated within the field of decoupled aquaponic 
systems (DAS). Arguments from researchers within this 
field are regularly made in terms of the inferred advantage 
of the decoupled aquaponic approach to plant production 
outcomes (Delaide et al., 2016; Goddek et al., 2016; Suhl 
et al., 2016; Monsees et al., 2017; Goddek & Vermeulen, 
2018; Suhl et al., 2018; Delaide et al., 2019; Eck et al., 
2019; Goddek et al., 2019; Blanchard et al., 2020). 
However, as the outcomes of the current research 
demonstrates, there was only one identified study that 
directly compared fully recirculating with decoupled 
aquaponics (with fish present in the direct hydraulic link 
between the fish and plant components) and that one study 
applied no replication (Monsees et al., 2017). The applied 
inference was that the decoupled approach performed 
better than the fully recirculating approach, as based on 
total fruit yields, but no mean results, no standard 
deviations or standard errors and no comparative statistical 
analyses were reported (Monsees et al., 2017). Of the 
seven articles that applied a RAS water or RAS water 
variation (e.g. complimented with additional nutrients) to a 
plant component, five argued that the decoupled design(s) 
they tested were equal to, or superior to, hydroponic 
controls or fully recirculating aquaponic analogues. 
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However, again, no replication was present in any of these 
studies and therefore, the inferred statistical differences 
observed were inappropriate and unreliable (Delaide et al., 
2016; Suhl et al., 2016; Goddek & Vermeulen, 2018; Suhl 
et al., 2018; Delaide et al., 2019). Therefore, are the 
arguments being made that decoupled aquaponic designs 
are better for commercial application than fully 
recirculating aquaponic designs (Delaide et al., 2016; 
Goddek et al., 2016; Suhl et al., 2016; Monsees et al., 
2017; Goddek & Vermeulen, 2018; Suhl et al., 2018; 
Delaide et al., 2019; Eck et al., 2019; Goddek et al., 2019; 
Blanchard et al., 2020) supported by scientific data 
generated from correct experimental designs and 
appropriately applied, comparative statistical analyses, or a 
perception generated from theoretical arguments? 

Experimental Replication Decision Matrix: 

An Experimental Replication Decision Matrix was 
developed to try and direct future researchers towards the 
application of correct replication within overall 
experimental design. Determining the comparison that is 
being performed within the overall experimental design is 
of paramount importance and assists to provide a broad 
understanding of what is being tested and assists to identify 
the comparison context (Hulbert, 1984; Quin & Keough, 
2002 ). 

This is generally associated with the basic question: What 
is being compared? 

Asking what is being compared in an overall experimental 
design context is not associated with identifying what 
parameters will be measured to determine any 
hypothesised differences (e.g. water chemistry - pH, D.O., 
EC, specific nutrient concentrations, etc.; plant growth or 
production - plant weight, plant length, leaf area, yield, 
etc.; fish growth - SGR, FCR, yield, etc.). It is about 
identifying what variable is being compared within the 
experimental design (e.g. aquaponic v hydroponic, fish 
species variations, fish density variations, fish feeding 
variations, fish to plant ratio variations, etc.). 

Identifying the comparison context assists in determining 
the final experimental context which directly leads to what 
aspect or component of the experimental set-up requires 
the replication. 

The important question is: Can what is being compared be 
compartmentalised as a sub-set of the entire culture 
system? 

If it is impossible to compartmentalise what is being 
compared as a sub-set of the entire culture system 
(aquaponic or hydroponic; coupled or decoupled), then the 


experiment is occurring within an aquaponic system (or 
any other culture system) context and therefore, correct 
experimental design requires entire culture system 
replication. 

Figure 1 shows there are relatively few comparisons where 
replication of only a sub-set of the entire system is valid, 
and these appear to be almost all related to comparisons 
associated with a plant component context (hydroponic 
component type comparisons - DWC, NFT, Media; plant 
growing substrate or media comparisons; direct plant foliar 
spray comparisons - nutrient, pesticides, etc.) or a direct 
plant mechanical treatment or similar comparison (pruning, 
leaf removal, growing tip excision, etc.). It is also 
important to understand that in these plant component 
context cases, replication of the component that provides 
the nutrient solution to the plants (either RAS/fish 
component of an aquaponic system, or sump/nutrient tank 
component of a irrigated nutrient solution) should not be 
applied, because this can then confuse which variable is 
actually effecting the plant growth outcome; the plant 
associated variable (e.g. media or substrate; hydroponic 
technology - NFT, DWC, media; foliar spray applied, etc.) 
or a fish component variable (fish, fish tank, filtration, 
etc.). Tlusty (2010) provides an analysis of an analogous 
situation that explains this difference in a RAS fish diet 
evaluation context where several different diets were fed to 
fish when replication was applied to the entire culture 
system (RAS) rather than the feed treatments (Arockiaraj 
& Applebaum, 2010). This study demonstrates that it was 
impossible to determine if it was the diet treatments or the 
individual filtration systems that caused the differences 
observed (Tlusty, 2010). 

In most experimental situations, entire system context 
replication will be appropriate. This is because most 
aquaponic studies attempt to differentiate the aquaponic 
system(s) they are testing; differentiation of one aquaponic 
system from another (e.g. coupled v decoupled aquaponic 
designs) (e.g. Monsees et al., 2017) or differentiation of an 
aquaponic system from a different system type (e.g. 
aquaponic v hydroponic v soil) (e.g. Graber & Junge, 
2009; Blidariu et al., 2013; Wilson et al., 2017; Jordan et 
al., 2018;). In these cases, at least three replicates of the 
entire aquaponic system and any control system (e.g. 
hydroponic system) would be required. 

Some situations will involve using nutrient solutions (no 
fish; solutions arising from separated RAS, nutrient- 
complimented RAS solutions and controls), which does 
differentiate them from entire aquaponic system 
comparisons, but in general, entire system replication will 
also be appropriate in these cases because the nutrient 
solution is being compared (e.g. Delaide et al., 2016; 
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Goddek & Vermeulen, 2018; Delaide et al., 2019; 
Blanchard et al., 2020). Again, in these cases, at least three 
replicates of each treatment system would be required (i.e. 
three nutrient sumps connected to three, independent plant 
culture devices for each treatment).One of these example 
studies, that compared nutrient solutions (hydroponic 
control vs nutrient complimented RAS water), rather than a 
complete aquaponic system, acknowledged that the 
experimental design applied (only one replicate system per 
treatment) did not meet the requirements of valid 
replication (Goddek & Vermeulen, 2018). 

For most aquaponic experiments (true aquaponic or RAS- 
derived nutrient solution variations), a good default 
experimental design is to adopt at least three replicates per 
treatment. A key point is that, avoidance of replicating 
entire aquaponic culture systems or units, as well as any 
control culture system (e.g. hydroponic system or unit), is 
not a good practice in an experimental design, replication 
or statistical analysis context for the majority of aquaponic 
comparative studies. 

The real difference lies with experiments that isolate 
specific, plant-associated effects (e.g. hydroponic type 
comparisons - DWC v NFT v Media, etc.; plant media or 
substrate comparisons; direct plant foliar spray 
comparisons, etc.). In these cases, one aquaponic system 
may be acceptable (and may be required), but replication 
of the experimental component (usually the plant 
component in plant growth studies) will be required and in 
general, it is recommended that at least three replicates of 
the plant-holding component will be required (e.g. if 
comparing NFT v DWC, at least three NFT channels and at 
least three DWC beds will be required; if comparing plant¬ 
growing media in NFT aquaponic culture, at least three 
separate NFT channels will be required). Salam et al. 
(2014) compared several different plant growing media 
(gravel, crushed bricks & a mixture of saw dust and gravel) 
and used a common fish component to feed the plant 
growing beds containing the different media, suggesting a 
correct starting experimental design for the type of 
comparison being performed. However, only two replicates 
per treatment were applied and while two is the minimum 
number of replicates required by definition (Gupta et al., 

2015) , three or more replicates are usually recommended 
for statistically valid crop comparisons (Roller et al., 

2016) . 


Example Studies Highlighting Replication Applications: 

1. Recirculating (Coupled) Aquaponics v 
Hydroponics - example system context 
comparisons: 

Alcarraz et al., (2018) compared a standard, 
recirculating, deep flow (DWC or raft) 
hydroponic system to a recirculating (coupled), 
deep flow (DWC or raft) aquaponic system using 
rainbow trout (Oncorhynchus mykiss ) juveniles 
(40 fish per replicate) that produced wastes used 
to provide nutrition to the plants. The experiment 
consisted of two treatments (hydroponic and 
aquaponic) and each treatment had three 
replicates. The replicates were complete system 
repetitions (i.e. three independent hydroponic 
systems and three independent aquaponic 
systems). The plants cultured were lettuce 
(Lactuca sativa L.) and each replicate contained 
thirty plants. The two culture systems were 
statistically compared to each other using 
ANOVA. This study was a good example of 
applying correct replication when comparing 
aquaponics to hydroponics. The culture system 
(hydroponic or aquaponic) was the variable being 
compared and therefore, replication of the entire 
culture system (or unit) was the valid approach. 
This meant that the statistical analysis applied to 
identify any difference between the two systems 
via the parameter compared (lettuce yield - gfwm' 
2 ) was valid and reliable. 

Lennard & Ward, (2019) compared a number of 
lettuce varieties (Lactuca sativa L.) and herbs (dill 
- Anethum graveolens L., rocket - Eruca sativa, 
coriander - Coriandrum sativum L. and parsley - 
Petroselinum crispum ) grown in a Nutrient Film 
Technique (NFT) hydroponic system and a NFT 
aquaponic system using Grass Carp 
(Ctenopharyngodon idella) to produce wastes 
used to provide nutrition to the plants. This study 
compared one semi-commercial-scale hydroponic 
system (1,800 plant spaces) to one semi- 
commercial-scale aquaponic system (1,800 plant 
spaces) and therefore, did not apply valid 
replication that would allow statistical 
comparison. The authors recognised the study was 
a crop production trial without replication of the 
test variable (i.e. the culture systems) and 
therefore, did not apply any statistical analysis 
and only reported mean outcomes with standard 
errors and percentage differences. However, they 
did argue the calculated percentage differences in 
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plant yields could be used as a measure of 
difference and argued that in the majority of 
cases, the aquaponic treatment outperformed the 
hydroponic treatment. This study was performed 
in a greenhouse and would be considered a crop 
or system demonstration trial. In a trial context it 
is not valid to infer differences based on the 
parameters measured in the absence of replication, 
but trends may be highlighted (Koller et al., 
2016). This study was an example of correct 
identification of the lack of replication and 
therefore, any statistical analysis would not have 
been valid. However, arguments that identified 
one culture systems superiority over the other, 
based on the observed percentage differences in 
plant yields, are worth scrutiny and not 
scientifically reliable (Koller et al., 2016). 

2. Aquaponic Solution v Hydroponic Solution (no 
fish) - example solution context comparisons: 

Nicoletto et al., (2018) compared three different 
nutrient solutions; water from an operating 
recirculating aquaponics system, the same 
aquaponic water complimented with phosphorous, 
potassium and a micro-nutrient mixture and a 
hydroponic nutrient solution control. Each 
treatment was replicated three times; therefore, a 
total of nine separate, independent culture systems 
or units were used. Each culture system consisted 
of a nutrient tank attached to four NFT channels, 
with a pump pumping the solution from the 
nutrient tank to the channels with a gravity return. 
The plants tested for growth parameters (plant 
height, yield, dry matter) were rocket ( Eruca 
vesicaria R) and mizuna ( Brassica rapa L. spp. 
Nipposinica M). None of the culture systems 
tested contained fish. This study was a good 
example of applying correct replication when 
comparing aquaponics nutrient solutions to a 
hydroponics nutrient solution. Even though fish 
were not present in any system, and even though 
only nutrient solutions were compared, the 
experiment was still comparing different culture 
systems (hydroponic and aquaponic) and 
therefore, replication of the entire system (unit) 
was the appropriate and valid approach. This 
meant that the statistical analysis applied to 
identify any difference between the two systems 
via the parameters compared was valid and 
reliable. 

Goddek & Vermeulen, (2018) compared 
aquaculture (RAS) nutrient complimented water 


with a hydroponic nutrient solution in a similar 
experimental culture system set-up to Nicoletto et 
al., (2018) (nutrient solution tanks attached to 
NFT channel arrays). They used only one nutrient 
tank per treatment and therefore, did not apply 
any replication to the two treatments. They 
identified that replication was not present within 
the experimental design (“Hydrologically 
speaking, this approach, however, cannot be 
considered as a repetition.”). However, despite the 
acknowledgement of the lack of replication, they 
still applied a statistical analysis (ANOVA) to the 
identified plant growth parameters measured 
(plant wet and dry weights) and used these 
statistical results to argue that one system 
(complimented RAS water) was better than the 
other (hydroponic nutrient solution). This study 
was an example of inappropriate statistics being 
applied in a situation with no replication 
(essentially, a trial) and using the identified 
statistical differences to infer the superiority of 
one approach over the other, was not a valid 
approach. 

3. Three Plant Growing Media Compared in One 
Aquaponic System - example plant component 
context 

Salam et al., (2014) compared three different plant 
growing media (gravel, crushed brick and a 
sawdust/gravel mixture) in an aquaponics system. 
The experimental design utilised a single fish tank 
attached to six separate plant grow beds (two beds 
for each different media), which therefore, 
realised two replicates per treatment tested. 
Because the “system” was not being compared or 
tested, but the plant growing media was, the use 
of a single fish tank feeding all the plant grow 
beds was appropriate, because this design 
removed the possibility of the fish tank or fish 
culture component being a variable, and 
concentrated any detected differences onto the 
media. Two replicates are considered the lowest 
number for valid replication (Gomez & Gomez, 
1984; Gupta et al., 2015) and therefore, the 
replication applied in this study was theoretically 
acceptable. This study is an example of good 
experimental design and appropriate application 
of replication for aquaponic studies concentrating 
on plant-associated component variables. 
However, it is suggested that three replicates 
would be more robust. 
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V. CONCLUSION 

It is concerning that aquaponic researchers are not applying 
correct replication to their experimental designs and then 
making inferences based on incorrectly applied statistical 
analyses and outcomes. It is more concerning that these 
studies are being published in peer reviewed journals, 
which are supposed to be providing a process to expertly 
review and ensure the validity and appropriateness of the 
experimental designs and statistics applied of the studies 
they publish. This suggests, that a lack of understanding of 
correct experimental design, correct application of 
replication and correct application of appropriate statistical 
analysis is far too common within the cohort of journals 
publishing aquaponic articles and among the associated 
reviewers assigned by these journals; a situation that is 
seen at similar levels within the associated disciplines of 
aquaculture (Hulbert, 1984; Ruohonen, 1998; Araujo, 
2008; Thorarensen et al., 2015) and crop science (Kramer 
et al., 2016; Raudonius, 2017). An experimental replication 
decision matrix was developed, and several example 
articles discussed, in an attempt to assist future aquaponic 
researchers to determine the replication requirements of the 
aquaponic studies they design. 
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